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I)i :.cu:*;i;«^d i\ro tour Jit toront approaches to tho 
'Valuation ot tho i icp lo ra<^ n t a t i on ' o f innovative couri^or,, Tho first 
approaiTh ir that oi Alkin and hii; aiisociatoc UCLA, This qroup 
<wpha:;i;:ra t hr uwd to colli^ct inforoatibn on tho doqroo to which an 
inviovativi^ course is i m j 1 <^mon totl for tho purpose of takinq corroctivo 
ai'ti'on to achievi^ lull installation hefalr prrforoancc dat^a are 
c^llectni. Thr :;ocond approach ir. ai>r.ociatod with tlio work of 
:>tallinijy, who limited the .'valuation to classrooms in which tho key 
• elements of an innovc\tion actually had bvvw implemontod by classroom 
*^.acliors. Tho third approach omerqed from the work ol Hall and 
IvMick:^. It suqqests t-iqht levels of use o'f an i Bplomo nta tion • The 
rouith approach is the approach suqqested by t !io author to maximize 
*h» mtormation from exp*rinontal i bp lomr n ta t ion cf innovative 
t'ourse::^ It req^iirer. comnli^tion of eleven steps. Assumptions^ 
-? : r.adv ant aqes and advantaoe.- of this approach are also discTissod, 
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u ..vuduM-; .,,ul ov.iluatoi-s hav,> inc.v.is inq)y c;cw to riMlizo Lh.iLnMny 

, "">nt,aMon ot I iu> innovat.iVe coiirs.^s ho'toiv acin pvoiiuMit, i inoasured Ono 
cons,M,u.Mu:o has lu..M> a t t,r„ Li on hy ros.arcl.Vr.; totho p.'oh 1 nns ' a s soc , a Lod , 
w Ui thr iinpl,M:,on(,it.ion ot innovations. Ainoiu, ovaluaLors at least tinv'o 
'II t(ot-<Mit niiproac lios. to trM> [iroliloiu hav- omorqod. . ■ " 

oarl iost of- tho thror stons from th,^ s ys toms -or i^n Led evalua.tors ' : 
^uc:l, as tnoso at lliLA who.n Alkin was Mu>: directarn^f tlu« Center for' the v 
Miidy of Lva nation. . lliis qroup emplias i .-cd the n,>ed- to collect mforma- 
MQii on the de>KtH> to v;hicli an innovative coiir^^e js implemented for tlie 
Puipose ot takm,, corrective action to acliieve fij-l 1 installation before 
''•;';''"'"'^";^-;\''''"'V''.' ^ V'^l'^^t^^l- AUhouqh it fs acknowledged -that modifi- 
^ oriqmal plan may he reiiuirej, this ap|iroach makes tiie 

u'l l ,n V,"^ MlLimately it will he possible to achieve un-iform . 

i"l Icmentation re.jardless of the number and types of tea*<?N, ^nOolved 
^'-id It requires repet 1 1 ion .\f the innovation until satisfactory implementa- 
tion has been acni(>v(\l . • ' . 

■ A second approacii^ is- p-rliaps most" closely associated with Re^car'cli for 
iH'ttei; ^chools ind particularly with the work of Stallmqs, who required ' 
curriculum deve op.h'S to specify key elements of an innovation, then 

ii-ntod Che evaluation to classrooms in wliic^i tlie key elements actually 
h.id been im,nl e,i;ented tiy classroom teachers. • Thits , the poEsibility 'that 
he acnievement, measured was not a cons-oquence of the innovation was mini.l 
; I ■ ^ '"'-^^l^' ^ u-ainmq and .supplvmq a large number of • 

teachers witli tne expectation of u5inq data from only a small number, and 
It u-|uire.s yes -no decisions concern i nq ;the presence or absence of an 
uinovavLion. T-he method would hecomq. ex J reme 1 y ...ci(|iens 1 ve and cumbersomp 
OS the numeerof key elements in an innovative course increases. - 

1 

A third aporoacli is emerninq from the work of Hall and Loucks at the " 
lexas Research and Development Cen'ter, wtiere the notion of measurinq the ■ ' 
relationship between student achievement and" e i giif " 1 eve 1 s " 0 f use" of an" 
imp omenta, ion is bemq deve 1 o.-ied . ' Th^s approach seems the mos t -pronlTs 1 ng 
ot tne tliree; but the suqg.ested scales are qcin?ral and abs-tract, and thus ' 
unlikelv to ada;it easily and usefully to pavttcular programs Further 
the approach dees not nr<jvTde mformatio-! for the three major purposes 'for 
wnicfi It useful to collect i.siplementdt i on information 
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llK? rit'St of thoso purposes, of (iroaLost inLorost Lo scliool adinini straLot^s , 
lUul tlie ono which the UCLA (jroup luul in iiii-tKl, is lo provide information 
tliat will ^!UTble aciininistratot^s to take wluitover correct4o7o action is 
necessary to'innre fully implement tlio iiuiovati vc course, Tlie second of 
tlitse purpoSL^s; probably of (jroo'test interest to [larents and taxpayers, 
is to [)rovide accurate measurements of wluit an innovative pt^ograiii can 
Mielp studervts to aclnevo, .Tlie tliird [mrpos.e, of n^^oatest intorest ;to 
cut;riculum developers, is to develr)p a fund of general izc]ble' knowledge 
useful ^"tlie de.ve roimrent of future innovative courses. 

The assum|)t1ons ^ str^ngtlis and \vectT<nosse's of tlie three .approaclies dis- 
cussed above suggest a'n approach to implementation, evaluation designed to 
meet all t.hree purposes and tlujs to maximize tlie' information from experi- * 
ntontal iinplc'n^tMita t ioii„ of ' innov^lt i ve cout^ses. Tin s ^appt^oacli requires com- * 
pletion of the steps discussed below, 

^ • ' yPJl^'^Iy^A oleiiients of__tlie jnnovation. While any educational 
program includ'es a large number of eTeiiTents," the number that. make it inno- 
'Vative and that^d^stingui sh it from other |>rograms wi 1 1 be rje^atively 
small. The initial ste(i in the afiproach boing suggested is to require 
ident Vf 1 ca tioti of th«e key eJenients. 

2'. Determi ne a scale and iiiethod_ fo^r measujf^iiKj_cacJi_key elemenjt. The 
choice between notuinai, ordinal, interval' an~d ratTo scaTes^"arid between 
such methods for determining thq extent to which the key element is present 
as obsot^va ti on , interviews or qjjestiQnnaires should be made based on 
characteristics of each key element, 

^- L^^JlL^'Jy^.LJ.^l^.^J.^ ij^ieiilgjltatjon Q.f each of tfie K elemen ts.'^? One 
might consider the 'e ight Tevel s proposed by llalland LoTjcks , ar's"oiiie other 
'number, but it probably is necessary on,ly to determine the .scores on each 
scale required fiKiJistinguish high, moderate and low implementation of the 
evaluation, or s imi 1 aJt^^fa i rl y gross categories: 

4. Pe|_ej^mj^r;i^G JL^I^^^ L^^^l^Jl^^_^il JJl'^_9Jj^^^_^ilL^'JL!L^ A^lli^i • Assume 

that eaoh teacher wi-lTctioose to implement ea-ch of the K elements at one 
of the L levels in(ile,pendently of one another. Identify the element with 
the largest number of 1 evel ^ of implementation and niulti|)ly this by the 
minimum numbeV of teachers (probably 5) ttiat you assume as necessary for 
statistical purpo^'es. in each cell. 

*5, S_eJ_ec_^_a ^e_t_yji3 jr^^yvh^ The main 

requirement Is a s i ngl e ■ educa t ionaU'a^ml ni s tra t i ve unit with a sufficient 
number of teachers of the required type. Other requirements including 
willingness to participate will also have to be met but need not concern us 
here. - " ^ • 

6, Tra in te acfiers to implement- the eva-^uation: The importance of involving 
teachers in the planning and implementation of any i nnova ti ve,,:.<:ourse has 
emerged from research in this area. Tlius, one shoul d <iXP^c t each individual 
teacher to make unpredictable decisions as to how the innovation will be 
adapted in each classroom. Hawever, and this is the key point, this approach 
permits, expects and even ehcourages each teacher to niake^ a fully independent 
decision on the level at which each element of th^ innovation , ynd particular 
the key elements, will.be implemented. Teachers should, however, be required 
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to record t),eir iiUenLion wi th ' re'spec t to iniplcmcntcUion of the K ol.emonLs 
in^torms of the sca-le^for each (step 2) or the. more gross CcUociories 

lun -i^^^^^^^ appropriate points at" which 

the impy.„enta ion of each of the K elencn-ts will be measured, using the 
cales designed m^step 2. Where teachers have been unable to reach the 
r-oaT tnT^i' • cU^Ssrooms, administrators should be 

uossib o f^'°^^^^\'?^ditional. rcsou rce^'Oci reach tifescv l.evel s as' quickly as 
• from tie begimil'j' ' contingency fund n,u^t have been budgeted . ' 

hlMLujlc, the ac^^^^^^^^^^ of each child. Use a ' cri ter iort-referenced 

fro,r;rrM\!s^r''^'v^T 

t": Jo apj degree' ^'^^^'^-^^^VP-ticipating- teacher Installed the i^nova- 

^^J'^i-^-'f -^f-^-^^ "^^^ to. achievement made by each 

kexelejierjt. The. data on degree of i^nf/l^TaTToTinTf e^T,n^oTel>^^^ 
rnV^Ti l^^. a'i^->lyzcd through such methods as nluitiple 

cot.elations and multiple regression using achievement Scores as the 
fi^'l:;^::^^^' obtam est;in,ates.f„the^effect of each key ^fement on 

rnMHn!^fn^H-'^^-"^-n^^^^^^ analysGS-of the data may be 

conducted to explore possible differences in the" -importance of each Jcey ' 
e ement^for students differing on sucn grou^drr as socioeconomic status! 
GthnicUy or academic track. The intent to conduct these analyses must be 

ttl .T. 'V^^^'Y^'"' ^'^'y t^^^" ^"to account when selecting 

the setting iVu'hich the innovation will be Implemented or during measure- 
ment of StudentAac-hieveinent. - •■ ^ , - » 

J^-,,,,i"-^^^-i.Jf-lil!iVU^^^ iiliRleinentation of the innovative 

5P.uts_e. ^ Wri tei.the -impTementation evaluation repor t Hue7p??tlliFTlTe 

probable relative contribution of each key element to achievement of ^ach 
type or student. ' 

The approach advocaj^d atJove involves at least two importast assumptions: 

1. That- 'teachers wi-l^I choose different levels at wh i ch Vhev^''^ 11 implement 
the various^ elem.ents, of an innovative course. ^ 

2. That teachers will choose implementation levels independent! y of one 
anoth^er.' .v_ . ^ " ^ ., 

/ .■^ ' 

The approacii advocated has lit least two importaiij;- di sadvantages ; 

1. It requires a fa i rl y ' subs tant i a 1 pilot e.ffort--probably a-minimuii; of ' 
'ib teacher^s and, tnerefore, in the IJ.S./about 500 elementary schoot 

oV- a larger, number of secondary students. 

2. It"is complexes technical and stik^'little more than an idea and 

■ therefore results almost inevitably will be difficult to interpret and 
anib.tguoti^. " ,■/ • ^ 
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V Conipt-nscitincj for 'those^|S^iitcigcs nee some very iinportnftt advnntaqes 

1. floro. often than not l^^^fii iiiium numhor of teachers suggested above 
will also^be tlie nuix iiiiiiiii •niiii)t)?;r required. 




The expectation of diWtjrent levels of implWtUation eliminates the 
need for a control group, and thus for many difficuUids associate'd - / 
wiJi this type of research, including, randomization , the assumption^ 7 
^at treatment and co^^t'rol groups liave^he same educdtional pbjec- ■ . V 
tives and control 1 ing cont.imination of treatment and control qroiips ' Y 

while n)aiuta.ining equivalency along such, d imens ions as socio-'- 
econoiDiX status. - . ^ , ■ 



i 
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3. It is more realistic to take advantage of natural variation among 
teachers than to ij^ a model that requires all teachers to implement 
) complex programs in the' same way. 
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t. Useful data is obtained regardless of the degree to which the 
vative^surse is implemented by each teaclier. 

> \ . - 

5. Estii'^tes can be made mathematica^l'y of tlie ef feels '.o f d i fferent 
degreesjof. implementation of the innovative course". ' Th i s can be done 
for-spedific types of students if proper care is taken in designinq 
the pilot impletii^entation. » 

' t 

6. There is no need to continually repeat the pilot implementation to 
obtain needed information. 

7. InfTJrmation can be obtained that'-meets /hq needs of a variety of 
audieflces who nd^qht be interested iiq v^^ious aspects of the innt)- 
vation. ^ 
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